For many years, a primary method for assessing the cognitive abilities of individuals has been by means of paper and pencil tests. However, paper and pencil tests have limited flexibility, do not present realistic ways of responding, do not provide immediate feedback of an examinee""s performance, and are costly with respect to scoring time and personnel requirements to administer them. Additionally, many situations exist in which such tests are inappropriate or have severe limitations for assessing higher level cognitive processes. There is a need, for example, for better methods of assessment of the cognitive and metacognitive skills required by managers and executives, especially those at the upper levels of organizations.
Such assessments are needed to identify executive leadership potential in assessment and promotion of managers and as a basis for executive development and training programs. Until recently, these skills were not fully identified or sufficiently validated. Also, measurement of such skills has been limited in the kinds and variety of responses appropriate to high level strategic decision making and problem solving. Recent research has identified the cognitive and metacognitive skills related to effective leadership in complex organizations, for example general problem solving, planning and implementation, solution construction, solution evaluation, social judgment and metacognitive processes. See, for example, Development And Evaluation Of Cognitive And Metacognitive Measures For Predicting Leadership Potential, Joanne C. Marshall-Mies and Edwin A. Fleishman et al., Leadership Quarterly, 11(1), 135-153. However, no testing methods and computerized means for measuring these cognitive and metacognitive skills related to effective leadership are available. What are needed are a computerized testing device for and method of assessing cognitive and metacognitive capabilities.
The objective of this invention is to provide a computer-based cognitive and metacognitive skill assessment and development tool suitable for implementation in supporting executive development. Current research on performance at mid and top levels of large scale organizations strongly suggests that cognitive and metacognitive skills are required for successful executive-level performance. Cognitive conceptual (integrative) skills are very important. However, the cognitive skill of self-awareness (metacognition) is critical, in that it involves the self-generated feedback and iterative evaluation process used in complex problem solving and decision-making.
The invention of the Metacognitive Leadership Exercises (MLE) assesses cognitive and metacognitive skills used in unstructured problem solving skills central to leader effectiveness. This is the primary reason for the existence of the invention. Use of this invention will provide critical information related to effective organizational management requirements.
The first task is to design the MLE measures to be appropriate for the proposed management population and to focus on strategic leadership and decision-making. Use of scenarios, prompts, responses, and a unique scoring methodology resulted in a computer-based assessment battery including six metacognitive measures (General Problem Solving, Planning and Implementation, Solution Construction, Solution Evaluation, Social Judgment, and Metacognitive Process) and a background questionnaire.
The second task was to demonstrate the reliability and construct- and criterion-related validity of the MLE measures. This involves an experimental study relating the MLE scores, a performance criterion measure, measures of related constructs (correlates), and collecting and analyzing the data. The study evaluated six MLE measures and a background questionnaire, a performance criterion, and three sets of correlates, i.e., the Modified Career Path Appreciation Scale or MCPA, seven MANSPEC variables, and four Myers-Briggs Type Indicator(copyright) (MBTI(copyright)) scores.
In using the MLE, the examiner is asked to participate via written materials and distribution of a disk which installs the MLE instructions and exercises, collects and scores the data, and installs the software. Participants complete the MLE using IBM-compatible computers with Microsoft Windows operating systems, in a job related validation study conducted at the National Defense University. Internal and external analyses are preferably conducted to determine the sample representativeness and the reliability and construct- and criterion-related validity of the MLE measures. The results are as follows.
Self-administration of the MLE on a variety of pcs and laptops running Windows and Unix was successful. Over 96 percent of the participants had no technical difficulties executing or completing the MLE measures, and over 94 percent made no comments or made positive comments about their experience with the MLE instructions and exercises.
The multiple-item MLE measures were reliable or internally consistent as evidenced by moderate to high alpha coefficients of 0.53 to 0.84 for Average Quality on the General Problem Solving, Social Judgment, and Metacognitive Process measures and 0.85 for Maximum Quality on the Metacognitive Process measure. The high level managerial population places severe time constraints on the data collection. Applying the Spearman-Brown correction formula, it is estimated that reliability would be increased significantly if the measures were doubled, i.e., to a range of 0.70 to 0.91 for General Problem Solving and Social Judgment and to a range of 0.89 to 96 for the highly reliable Metacognitive Process measure. These results reveal that individual differences in scores obtained by high level executives produce consistent results on these measures of metacognitive skill.
The MLE score distributions indicate that senior level individuals differ in performance on the MLE measures; the large standard deviations indicated a considerable range of scores for any given measure. These findings, especially given the unique sample, confirmed that the MLE scores reflect individual differences in the skills measured. Also, relatively low to moderate inter-correlations among component MLE measure scores confirmed partial independence of the measures and provided evidence that the measures tap different dimensions of problem solving associated with leadership skills.
MLE measures are slightly to moderately related to several correlates, showing a convergence between the constructs measured. Of particular interest is the significant but moderate correlation of General Problem Solving and Social Judgment with the MCPA, a work sample which taps work disposition as articulated by the subject and judged by the interviewer and which shows how discretion is exercised and how the person copes with complexity. It purports to measure fluid intelligence, concept formation, insight/intuition, comfort with ambiguity/uncertainty, and characteristic strategies and feelings about work. Also of interest is the correlation of General Problem Solving and Social Judgment with Analytical Skills, a Holland Vocational Preference Inventory score which indicates a preference for using one""s analytical and problem solving abilities and for using intelligence to achieve ends.
The MLE General Problem Solving and Social Judgment measures provide positive correlations with MCPA and Analytical Skills. Two multiple regression analyses are conducted using the MCPA as the dependent variable. The first is a simultaneous multiple regression with General Problem Solving and Social Judgment as independent variables. Together, the MLE measures show a multiple correlation of 0.38 (p less than 0.01), with General Problem Solving representing unique variance in predicting the MCPA. The second regression enters Analytical Skills first, followed by the simultaneous addition of the MLE scores. This results in a multiple correlation of 0.60 (p less than 0.01), with Analytical Skills representing unique variance in predicting the MCPA and General Problem Solving approaching but not reaching the level required to represent unique variance.
A Distinguished Graduate (DG) criterion of performance significantly correlates with two MLE measures, i.e., Solution Construction(copyright)=0.30, p less than 0.01) and Metacognitive Process(copyright)=0.40, p less than 0.01). These coefficients indicate a robust relationship between these MLE measures and the performance criterion. A simultaneous multiple regression was then performed with DG points as the dependent variable and the most valid Solution Construction and Metacognitive Process scores as independent variables. These two MLE measures showed a multiple correlation of 0.47 (p less than 0.01), with each measure representing unique variance in predicting the performance criterion.
The invention is used for the assessment and development of metacognitive skills. Metacognition can be reliably measured using several of the MLE performance measures. Two of the MLE measures are valid predictors of the class rank criterion (DG). The measurement of these important leadership skills focuses on evaluation of the MLE measures: (1) as measures of the effectiveness of training on metacognitive problem-solving skills in the area of strategic decision making; (2) as a screening tool for the early identification and development of problem-solving and decision-making skills, including metacognitive skills; and (3) to assess their validity against more objective, varied performance criteria and with varied populations. MLE includes tactical and operational as well as strategic-level decision-making and problem-solving.
This invention includes certain considerations in developing a computerized assessment of metacognitive skills and a description and justification for the design of the computerized Metacognitive Leadership Exercises (MLE) to measure metacognitive skills. The first section discusses issues related to the general characteristics of the assessments. The focus of the present invention is on the assessment of high-level cognitive skills, that is, metacognitive skills. Metacognitive skills are defined as high-level expertise in the conscious evaluation and regulation of problem construction, solution generation and implementation processes. The invention describes general design features to be included in the assessment instruments and shows how these general design features relate to the development of reliable and valid measures of metacognitive skills.
Traditional, trait-based psychometric assessment is concerned with ordering people with respect to their levels of particular, and often empirically defined, attributes or abilities. Examples of traditional assessments include the General Aptitude Test Battery (GATB) and the Scholastic Aptitude Test (SAT). Such measures are practical, effective tools for the selection issues they were designed to address. While effective at selecting and classifying people based on individual differences in level of performance, these measures are not designed to allow assessment of the cognitive processes or strategies people use in performance. Traditional measurement, consistent with the prevailing behaviorist tradition in which it was developed, treats the mind and its functioning as a xe2x80x9cblack box.xe2x80x9d The interest here is typically with performance based on stable, enduring behavioral characteristics.
In contrast, cognitive assessments are concerned with assessing how people know what they know, do what they can do, and the ways in which they can increase these capacities. To accomplish these goals, cognitive psychologists have attempted to assess the mind""s operations, or processes. Work in this area has both focused on lower-level, biologically linked automatic mental operations and higher-level, learned mental operations. Such cognitive processes have been classified as fast or slow, with the latter being the focus of the present invention. It is useful to contrast how researchers have investigated fast and slow cognitive processes to clarify the nature of metacognitive skills.
A good deal of work has gone into investigating fast cognitive processes. These processes operate automatically, beneath the level of awareness, and are generally measured by response latencies to task components, error rates, or direct measures of people""s eye movements and neurophysiology. The operation of fast, automatic cognitive processes, e.g., information encoding, is independent of domain-specific knowledge and skills. These fast, automatic processes are not directly observable. Therefore, their measurement depends on the extent to which the assessment design provides an adequate basis for inferring that hypothesized, unseen mental processes are contributing to observable indices of performance. Fast processes are measured in the laboratory with simple cognitive tasks such as analogies or spatial visualization items drawn from psychometric ability tests. Examples include the stimulus encoding, memory search-and-mapping, and evaluation operations comprising components of various problem solving models.
Leader cognitive skills targeted in the present invention are metacognitive skills. Metacognitive skills are slow process skills because, by definition, metacognition is subject to introspection, that is, awareness. Slow, complex skills require the use of prior knowledge, and include such activities as complex problem solving, monitoring of complex systems, etc. Measurement approaches for complex cognitive skills have reflected the finding that these skills are conscious, directly observable control processes. Accordingly, researchers have generally measured such complex cognitive skills by presenting people with tasks intended to elicit their application. This is identical to common assessment practices in education and industry, where the goal of assessment is to find out an individuals knowledge and skill with respect to some absolute standard. The design of the tasks may often only reflect a simple consensus of expert opinion as to their appropriateness and power to elicit the skill, that is, its face- and content-validity. In responding to the task, people provide open-ended verbal and behavioral records or retrospective interviews are used to record observations.
No matter whether automatic (fast) or controlled (slow) processes are the objects of assessment, cognitive assessments must be designed far more carefully than traditional psychometric measures. Their design must rely not only on empirical findings, but also on substantive theory, and the theory underlying the assessment design must specify (a) which cognitive operations contribute to performance, and (b) the sequence of these operations. Task components comprising the assessment must reflect links to components of the specified theoretical model at each stage of the design. Such integration provides the basis for establishing the construct validity of assessment of the unseen, automatic mental operations. Efforts to solve the inferential validity issues raised by cognitive research have led not only to experimentally-based methodologies for cognitive measurement, but also to the development of powerful new methodologies linking the experimental and psychometric approaches.
Tasks that are appropriate for assessing cognitive skills in general include any task designed to investigate how people perceive, understand, acquire, and use knowledge. Examples of such cognitive tasks range from school learning tasks to educational and psychological tests. What is crucial is that the tasks are designed so that the inferential statements resulting from the observations are strong, and speak to the validity of the assessment. That is, performance on the assessment should call forth the intended cognitive or metacognitive skill, and be related in the expected way to real-world performance.
There are other issues bearing on the design of the assessments. Such considerations are the cost and practicality of administering metacognitive skills assessments are particularly important in the present invention. In the following sections, a brief summary of current approaches to metacognitive skill assessment is presented, followed by a discussion of specific task design considerations bearing on the strength and validity of the inferences derived from Management Leadership Exercises. In particular, the focus is on presenting an alternative to open-ended responses that meet these criteria and offer a solution to the issues of practicality and cost.
Metacognitive Assessment
Researchers have devised several ways to assess metacognitive skills. These different approaches reflect the fact that metacognitive assessment should be tailored to fit the nature of the skills and the theoretical model posited to underlie performance. For example, a text-faulting procedure investigates metacognitive performance of adults while reading. The specific metacognitive skill investigated was the monitoring of reading comprehension. People were asked to detect faults in written material that could be of any one of seven different kinds. These faults ranged from superficial, such as spelling errors, to more abstract, such as incongruent semantic relationships. Planning skills were assessed via a computer task in which people were asked to travel to several locations displayed on the screen, starting from an established xe2x80x9chomexe2x80x9d location. Data obtained include the time between the initial presentation of the problem and initial move, and between the subsequent moves, and measures of the total distance traveled by the moves. A xe2x80x9cthink aloudxe2x80x9d methodology assessed planning in electronic troubleshooting tasks. Novice and expert troubleshooters were presented with a problem and asked to suggest hypotheses, state plans, and specify the steps they expected to take in solving the problem. Those protocols were mapped onto the replaceable electronic components people selected. This mapping revealed that experts used only relevant electronic components and their plans showed a systematic and linked transition in making these choices. In contrast, the mappings of many novices showed that they tended to use irrelevant components in a series of unsystematic and unrelated action steps. Assessment of metacognitive skill within a specific domain decomposed physics problems into a solution plan and an executed plan, each of which could be further decomposed into sub-activities of identifying a concept or procedure and justifying its selection. After identifying relevant principles and justifications for their application, the individual could then execute the plan and solve the problem. The assessment approach incorporated response option formats ranging from completely open-ended to fixed choice.
Several specific characteristics of these metacognitive skill assessment methodologies were used for the design of Metacognitive Leadership Exercises. First, all of the tasks described are complex tasks. As discussed later in more detail, complex tasks are required to adequately assess complex performance.
Second, the tasks are skill-specific. For example, planning skill is elicited by a task involving planning, and reading comprehension by one in which people read complex material. Thus, for tasks assessing high-level complex cognition, the face- and content-validity of the task must carry a good deal of the burden for helping to establish inferences that the targeted metacognitive skill is being elicited.
Third, a variety of response alternatives are incorporated, ranging from the content analysis of think-aloud protocols to time-on-task (as contrasted with response xe2x80x9clatencies,xe2x80x9d per se). The complexity, skill-specificity, and variety of responses that are characteristic of current measures of metacognitive skills underscore an important point: Complex cognition, occurring as it does at the level of awareness, does not require the elaborate response latency and error rate analysis typical of measures of unconscious cognitive processes. In fact, the meaning of pure latency variables, problematic as it is in studies of low-level cognition, is far more questionable for complex cognition. For example, a fast latency of response to a component of a complex task may reflect either the automatic application of highly elaborated, expert knowledge and skills, or lack of effort or inattention. We have no way of knowing which, given only the latency value. Instead of latencies, however, sound inferences regarding the application of the targeted high-level skills may be made by judiciously structuring the task and observing qualitative differences in the content of responses. It is this incorporation of qualitative assessment into the measurement design that helps capture the process application and strategic differences that reflect metacognitive skill. Nevertheless, a travel planning paradigm of time-on-task may be used to provide indices of complex metacognitive skills when its substantive meaning can be established.
The theoretical models underlying performance on the high-level tasks are more general, as prescribed by the conscious, observable nature of the targeted metacognitive skills. For example, component models of the fast process skills contributing to performance of analogical reasoning items specify sequential encoding, search and mapping, evaluation, and response operations. A tightly integrated set of testable hypotheses represented their application. In contrast, the examples of metacognitive assessments sketched out above provide no such explicit sequence of operations. Instead, inferences regarding the application and level of the targeted cognitive skills are made based on the nature of the task and accompanying responses.
MLE Design Criteria
MLE design criteria are described, focusing on six critical aspects: complexity, level of domain specificity, fidelity, response type, administration, and scoring.
Design of cognitive assessments"" items, structure, and response format represents the xe2x80x9cobservational design.xe2x80x9d The purpose of the observational design is to structure the observations available from the task so that defensible inferences about the nature of the targeted skill can be made from the observations. General factors bearing on the nature of the observational design of the MLE are elaborated.
The MLE measures were developed to have the following characteristics:
complexity
domain specificity
fidelity
construct-based, fixed response format/
Complexity. Assessments of complex cognitive skills require tasks. Complex tasks are distinguished from simple tasks by their requirement for integration of diverse knowledge and skills. Cognitive psychologists classify tasks requiring the integration of diverse knowledge and skills as unstructured, ill-defined tasks. Of course, complex cognition, such as introspection, may occur during the performance of simple tasks. Simple tasks, however, do not allow people the opportunity to express the complex cognitive skill in a sufficient variety of ways. Therefore, typical items from standard ability tests do not make informative cognitive tasks. Thus, the most useful information that may be obtained from cognitive assessment is detecting the important differences in the way people complete the assessments, that is, perform the tasks. Assessing metacognitive and other complex knowledge and skills incorporates complex exercises.
Complex tasks are useful in the design of cognitive assessments. First, complex tasks allow integration of more requisite performance dimensions. This provides a basis for better assessing qualitative differences between individuals on their capacity to integrate what they know for task performance. Second, complex tasks provide multiple xe2x80x9csnapshotsxe2x80x9d or ways of representing performance. The resulting larger set of potentially useful variables can provide the patterns of convergent and divergent relationships and interactions that will evidence the construct validity of the skill assessments. Third, complex tasks allow for the assessment of awareness. Evidence for awareness is important to establish that the targeted metacognitive skill is being applied. Finally, measures derived from performance on complex tasks are better at classifying individuals with respect to their stage of development. Qualitative assessments of complex tasks can thus allow assessments of current readiness for acquiring new knowledge, that is, future potential, thus providing an important link to the development of training interventions.
Domain specificity. Measures of metacognitive skills may be designed to reflect either a general or domain-specific focus. Metacognitive skills are not tied to specific domains, given the dependency of complex cognition on existing knowledge and skills, it is not surprising that others have accrued evidence that at least some metacognitive skills are domain-specific. An objective of this invention, however, is to assess metacognition in leaders. Domain-specific metacognitive skills may be both relevant to real-world performance and amenable to training. Needs exist to assess the metacognitive skills used by mid-level leaders in creative problem solving. Further, needs exist to assess the training feasibility of these skills as they are applied by leaders. Accordingly, the tasks in the example are constructed for assessments to reflect leadership content.
Designing tasks to reflect leadership content strengthens the design of the assessments in three ways. First, stronger inferences can be made from tasks with leadership content about the likelihood that leader task performance will generalize to real-world problems. This is because the tasks may take the form of a simulation. Second, stressing content and process enhances the likelihood that task performance reflects the targeted metacognitive skills for all individuals, that is, scores on the assessment will mean the same thing for all individuals. For example, some leaders in intensive training may be more likely to react negatively to non-leadership scenarios. Such tasks may seem irrelevant, because the tasks are perceived as unrelated to their training. For those who react negatively, task performance may reflect something other than the application of the targeted metacognitive skill. Finally, the use of leadership scenarios will elicit maximal performance from the examinees, as they may better use existing knowledge and skills in task performance.
Fidelity. An issue in the design of any assessment is its fidelity, or the extent to which it mimics a real-world task. There is a good deal of evidence that low-fidelity simulations might be more than adequate for eliciting requisite skills and permitting reliable observations. Specific examples of a class of viable low-fidelity tasks are represented by xe2x80x9ccontrolled simulationxe2x80x9d tasks. Controlled simulations are complex tasks that represent a compromise between rigid traditional tester-controlled observational settings and the wholly unstructured observational setting found in the real world.
The purpose of any simulation is to approximate a realistic situation so that one can generalize from simulation performance to real-world performance. Controlled simulations satisfy this ability to generalize criterion. Controlled simulations include the use of written scenarios. Controlled simulations have been used most prominently in the arena of medical education in patient management problems. For example, a simulated patient (via written or oral dialogue) presents the examinee with initial symptoms; the examinee requests tests, considers their results, prescribes treatments, and monitors their effects, generally attempting to identify and treat the initially unknown disease. The controlled simulation model of assessment allows identification of patterns of performance that might suggest specific associations among facts in examinees"" schema, or show the use of effective or ineffective problem-solving strategies.
Response format. Traditional simulations or similar assessments in education and industry, like those of complex cognition, generally rely on scoring or rating open-ended responses or protocols. Consequently, such assessments suffer from several methodological problems related to such scoring. Subjective evaluations are costly, time consuming, and subject to the effects of rater error. Although it is common practice to reduce error by training judges and providing illustrations of good and poor performance, judges do not always follow these rules and their evaluations may be influenced by a host of situational factors, such as stereotypic assumptions about the nature of acceptable performance. While judges may agree, this agreement may not reflect the targeted skills, especially when exercises have not been carefully designed to call forth these skills. Even in cases where the scoring of open-ended responses does not require judges, the procedures required for scoring and entering these data are labor intensive. This is true even when researchers are careful to structure and objectively analyze written or verbal protocols.
Problems associated with subjective scoring are exacerbated by task complexity. Not only does complexity make assessment expensive and scoring more difficult, but also it makes it difficult to obtain an adequate sampling of skill expression. Often, only a few exercises comprise any given session, of which only one or two may be targeted on specific skills. Such assessments are thought to yield performance scores reflecting exercise methods rather than the skill of concern, behaving like a traditional test comprising only one or two items. To complicate matters further, many exercises are interactive and occur in poorly-controlled settings. Thus, it is difficult to know how much of an individual""s score reflects aspects of the assessment setting versus the individual""s actual skill.
Traditional multiple choice formats, however, do not represent a useful alternative. First, even if responses to traditional 5-point multiple choice items required the application of the same processes as real-world tasks, such assessment would be less efficient than more integrated, complex responses. Second, traditional multiple choice items may actually elicit processes different from those required for doing real-world tasks. Using a criterion-related validity approach with such measures might lead to the identification of people who might be good at recognizing isolated facts. However, they may be poor at integrating the knowledge, skill, and strategies needed for complex tasks. Finally, multiple choice items simply look different from real-world tasks and are thus easily characterized as irrelevant and trivial, a face-validity consideration of particular import with executives. Given problems inherent in both traditional open-ended and multiple choice closed assessment formats, an alternative approach is used.
In assessment, the nature of the responses elicited should represent a performance; specifically, a performance that reflects application of the targeted skill. In assessment of high-level skills, researchers typically structure the task so that key components of task performance are elicited. Thus, for writing, one component of writing skillsxe2x80x94revisionxe2x80x94might be assessed by asking people how they would revise an essay. Another component of writing skillxe2x80x94planning might be assessed by asking people to outline an essay. As with classic, fixed-format methods, responses to these tasks are designed to elicit or reflect the use of attributes comprising the key components of the skill. Different attributes or characteristics of skilled performance are manifested in the resulting responses.
The MLE solution to the problems inherent in traditional fixed-format methods was to develop fixed response options for the tasks that represent qualitatively different expressions of the targeted skill. In responding to each prompt, examinees could select more than one response from several (apparently) equally attractive alternatives. An analysis of the responses would then be used to generate a profile of scores for each examinee based on the content of their most frequently selected items. Clearly, the nature of the response options is critical to the success of this strategy. The content of the response options must reflect important qualitative differences in the way people might complete the task. From these, inferences may be made regarding the processes being applied in task performance, and scores may be related to other indices of learning, development, and achievement.
The MLE strategy for development of the response options was to link them to a set of constructs or responses reflecting strategic differences in the application of domain knowledge and skills. As is discussed later, these linkages were made by developing response options in a series of focus groups consisting of subject matter experts. Systematic preferences in selecting responses were then held to reflect strategic differences in task performance. The critical evaluation and selection of equally plausible responses represent complex responding. The design of tasks to capture complex responses should be the real focus of assessment.
Further, related approaches to structuring response options have proven successful. When combined with convergent and divergent evidence from time-on-task and control measures, this measurement strategy provides a sound basis for inferring the application of the targeted metacognitive skill. Furthermore, this strategy provides a solution to important issues of practicality and cost of administration and scoring.
Linking fixed-response options to performance-relevant constructs represents a viable approach to construct measurement. When combined with the MLE strategy of allowing the selection of more than one response, the approach would conform to an important requirement of the assessment of complex skills: People should not be constrained to provide responses in which there is only one right, or wrong answer. The responses elicited, furthermore, reflect different strategies or approaches that can be used to perform a task, and could be linked to observable skills. Like traditional assessment exercises, the MLE elicits complex responses, providing a potentially more comprehensive description of the individual""s performance capabilities.
The specified MLE assessment design considerations serve to structure the observations available from each task. Specific design features incorporated into the MLE assessments include high complexity, domain specificity, a low level of fidelity, and the use of construct-based, fixed response formats. Some or all these design features are characteristic of current measures of high-level cognitive skills. Thus, the MLE measures incorporate features that allow defensible inferences to be made about the nature of the targeted metacognitive skills.